Rejoinder: Expert Elicitation for Reliable System Design

First of all, we would like to thank the discussants for the care and thoughtfulness that they have taken in preparing their comments. Koehler presents a helpful discussion, putting forward a number of different ideas that generalize the approach taken. A taxonomy for technical system elicitation would provide useful guidance for practitioners and serve to codify applicable assumptions during the different systems engineering phases. Although more research is needed here, one could see the emergence of international standards that rely on such a taxonomy. We acknowledge that the elicitation problem varies greatly depending on the technical system as pointed out by Koehler and we have sought to generalize our experience in studying complex systems, including aerospace, rail and naval for both commercial and defense markets. This explains our bias toward the “closed loop” case. We agree with the two extra areas of expert elicitation identified for “waterfall” cases: lack of expertise continuity and the problem of “forward casting” requirements for an existing system. Both of these relate to discontinuous changes in system operation. Such changes have occurred most obviously in military systems and other projects with long lead times. However, in the commercial world, such discontinuities can be forced by regulatory or market changes, or by outsourcing decisions. These may make historic data collection taxonomies less relevant to the reliability questions posed to support new operational decisions and, therefore, provide new areas of application for expert judgement techniques. The final point raised by Koehler about the difficulties imposed by system complexity is well made